Building “Stuff”?

Charles F. Vardeman II

Center for Research Computing, University of Notre Dame

2023-09-05

Building Stuff?

Building Agents based on Large Language Models!

Building “Stuff”…

Learn More: Effective testing for machine learning systems

Youtube Discussion: MLOps Chat: How Should We Test ML Models? with Data Scientist Jeremy Jordan

Building “Stuff”…

Learn More: Effective testing for machine learning systems

You are here ⬇️

What Kind of LLM Agents are we trying to build?

  • Conversational Agents
  • vs. Cognitive Autonomous Agents
  • vs. Agents tuned for a Data Processing Task

We will focus on Conversational Agents…

The Best Advice we can Give

Andrej Karpathy, “State of GPT” | BRK216HFS, Microsoft Build, 2023.

https://www.youtube.com/watch?v=bZQun8Y4L2A.

Andrej Karpathy, “State of GPT” | BRK216HFS, Microsoft Build, 2023.

https://www.youtube.com/watch?v=bZQun8Y4L2A.

Caveat: You are at the Edge of Research and Practice!

Instruction Prompting

(GPT-3) Instruct-GPT Reinforcement Learning from Human Feedback

Ouyang, Long, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, et al. 

“Training Language Models to Follow Instructions with Human Feedback.”

arXiv, March 4, 2022. https://doi.org/10.48550/arXiv.2203.02155.

Facilitates Conversational Agents to “Converse” in a Set Style!

(GPT-3) Large Language Models are Zero Shot Reasoners (Chain-of-Thought Reasoning)

Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.

“Large Language Models Are Zero-Shot Reasoners.” arXiv, January 29, 2023.

https://doi.org/10.48550/arXiv.2205.11916.

Tools for “Prompt Engineering”

Henry Zeng, Lauryn Gayhardt, Jill Grant “What is Azure Machine Learning prompt flow

(preview) - Azure Machine Learning,” Jul. 02, 2023.

http://tiny.cc/kelavz (accessed Sep. 04, 2023).

Tools for “Prompt Engineering”

Henry Zeng, Lauryn Gayhardt, Jill Grant “What is Azure Machine Learning prompt flow

(preview) - Azure Machine Learning,” Jul. 02, 2023.

http://tiny.cc/kelavz (accessed Sep. 04, 2023).

LLMs as Reasoners using Prompts!

Besta, Maciej, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, et al

“Graph of Thoughts: Solving Elaborate Problems with Large Language Models.” arXiv, August 21, 2023.

http://arxiv.org/abs/2308.09687.

Prompt Enginnering Guide

“Prompt Engineering Guide – Nextra.”

https://www.promptingguide.ai/ (accessed Sep. 04, 2023).

We want Large Language Models to be Factual!

  • “Fine Tuning”
  • Retrieval Augmented Generation (RAG)

We want Large Language Models to be Factual!

  • Fine-Tuning: augment the behavior of the model
  • Retrieval: introduce new knowledge to the model
  • Retreval Aware Training (RAT) Fine-tune the model to use or ignore retrieved content

LlamaIndex to Build Hybrid KGs

“Custom Retriever Combining KG Index and VectorStore Index

  • LlamaIndex 🦙 0.8.5.Post2.” Accessed August 22, 2023.

http://tiny.cc/oflavz.

Gorilla: Retrieval Aware Training for APIs

S. Patil, “Gorilla: Large Language Model Connected with Massive APIs [Project Website].” Sep. 04, 2023.

Accessed: Sep. 04, 2023. [Online]. Available: https://github.com/ShishirPatil/gorilla

Gorilla: Retrieval Aware Training for APIs

S. Patil, “Gorilla: Large Language Model Connected with Massive APIs [Project Website].” Sep. 04, 2023.

Accessed: Sep. 04, 2023. [Online]. Available: https://github.com/ShishirPatil/gorilla

“Big Models” vs “Small Models”

  • Models as a service (Bedrock, OpenAI API, Anthropic Claude)
    • Generally more difficult to “Fine-Tune” (GPT-3.5 turbo)1
    • Models are generally more capable (Factuality, Instructions, Reasoning)]
    • “Coin-operated” pay per/token
  • “Open License” 7B-70B Parameter Models
    • Mostly based on Meta AI LLama or LLama 2 models
    • Require more effort to work consistently
    • Models can run on reduced hardware requirements
    • Can be fine-tuned for task specific workflows

Small Models with custom grammar (llama.cpp)

JSON-Grammar

root   ::= object
value  ::= object | array | string | number | ("true" | "false" | "null") ws

object ::=
  "{" ws (
            string ":" ws value
    ("," ws string ":" ws value)*
  )? "}" ws

array  ::=
  "[" ws (
            value
    ("," ws value)*
  )? "]" ws

string ::=
  "\"" (
    [^"\\] |
    "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes
  )* "\"" ws

number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? ws

# Optional space: by convention, applied in this grammar after literal chars when allowed
ws ::= ([ \t\n] ws)?

“speculative : add grammar support by ggerganov · Pull Request #2991 · ggerganov/llama.cpp,”

GitHub. https://github.com/ggerganov/llama.cpp/pull/2991 (accessed Sep. 04, 2023).

“The state of GPT” Recommendations

Andrej Karpathy, “State of GPT” | BRK216HFS, Microsoft Build, 2023.

https://www.youtube.com/watch?v=bZQun8Y4L2A.

Open Source Community

Matt Bronstein and Rajko Radovanovic, “Supporting the Open Source AI Community,”

Andreessen Horowitz, Aug. 30, 2023.

http://tiny.cc/uflavz (accessed Sep. 03, 2023).

Today is the First Step on your Journey to Building LLM Based Applications!…